Repository navigation
Feat: automation dependencies - #2706
BelhsanHmida wants to merge 52 commits into
Conversation
…s end Context: - Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs. Change: - record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded. - The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job. - Plugin jobs carry automation_run_id in their trigger. - A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Schedule, report and plugin runs now record how their jobs ended. Change: - Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded. - A plugin run records its job's outcome, also when the job finished before the run recorded it. - Report run tests check the run's outcome: succeeded, nothing to do, and refused. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Schedule, report and plugin runs now record how their jobs ended, as forecast runs did. Change: - Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Schedule, report and plugin runs now record how their jobs ended, not only forecast runs. Change: - Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - main moved 11 commits on, including #2689, which saves what a data generator returns through generator_results and renames the unchecked-sensor errors to GeneratorWritesUncheckedSensor. Change: - Keep the record_outcome_on_automation_run imports next to main's new ones in data/automations.py and services/reporting.py. - Keep the decorator on make_schedule where main dropped ScheduleWritesUncheckedSensor. - The report refusal test expects GeneratorWritesUncheckedSensor as the run's error type. - The failed schedule test breaks the save through save_generator_results, which make_schedule now calls instead of save_to_db. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue. Change: - Say execution state and job records are kept for every type of automation. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - main moved 4 commits on, and #2647 changed the rq.job import in test_automations_fresh_db.py. Change: - Keep main's `from rq.job import Job, JobStatus` and this branch's `from sqlalchemy import select`. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Issue #2586: an automation consuming another's results could only be ordered by staggering cron strings Change: - New automation_dependency table, and 'waiting' and 'skipped' run states - A run waits for the upstream runs scheduled for the same time, and is dispatched once they ended, or skipped if one failed (unless failure is allowed) - Waiting runs are released when the upstream run ends, and by the runner's tick as a backstop - Dependencies are validated: same schedule, no cycles, readable upstream automations Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…copy Context: - Issue #2586 asks what a copy does with a dependency Change: - A copy waits for the copied upstream automation where that was copied too, and for the original one otherwise Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Issue #2586 Change: - depends-on on creating, updating and reading automations, and upstream-runs on run stats - PATCH checks the user may read the upstream automations, and refuses a schedule that breaks a dependency Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Issue #2586 Change: - add/edit automation take --depends-on and --allow-failure-of, edit also --no-dependencies - show automation lists dependencies, run-automations reports waiting and skipped runs Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Issue #2586 Change: - A 'Waits for' picker on the create and edit forms, copying the picked automation's schedule into the create form - The details panel lists the dependencies Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Issue #2586 Change: - Runner tests with a real worker: wait then release, skip, allow-failure, cascade, missed release, passed-over and inactive upstreams - Validation, API permission and schedule checks, CLI options, and copied dependencies Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Issue #2586 Change: - New section on dependencies, the waiting and skipped run states, and that a run on demand does not wait Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - PR #2706 Change: - Entry in 'Automations, in detail' and in the CLI changelog Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Resolve conflicts with the squash-merged #2670 and with #2657: take main's naming of schedule jobs and its job-failure path, keep the release of waiting runs where a run ends, and use AUTOMATION_RUN_RETRIED_TYPES, which now includes scheduling. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…tion bf22b21d0990 Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
A run now waits for each upstream automation's latest run due at or before its own scheduled time, by the upstream automation's own cron string and timezone, so for instance an hourly forecast can wait for meter data pulled every 15 minutes. The same-schedule check on creation and edit is dropped, and so is the UI copying the upstream schedule into the form. The run record names the time the upstream run was due at (due-at). Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
flexmeasures jobs run-automation --with-dependents, the trigger endpoint's with-dependents query parameter and the Automations page's Run with dependents start a run of the automation now, and a run for each active automation that waits for it, directly or through others, which waits for the runs started along with it, and is skipped if one of those failed. This is how a chain is re-run, for instance after backfilling the data at its start. A run records whether it was due by schedule or asked for on demand (trigger), so that on-demand runs can share a minute with scheduled runs, and a scheduled run only ever waits for scheduled upstream runs. The runner does not retry on-demand runs, as whoever started them is told when they fail. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
The details panel on the Automations page draws the automations listed there that this one waits for, on the left, and those that wait for it, on the right, directly or through others, with a dashed arrow where the waiting automation runs even when the one it waits for failed. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
The Waits for field in the creation and edit forms was a multiple select, which took Ctrl-clicks to pick several automations and cut long names off. Each automation is now a row with a tick box, naming its asset and recurrence, and dragging the corner of the list enlarges it. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
v1.1.0 shipped without this feature, so its changelog entries move to new sections after it. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Next to the chain of the automation being read, its details can now also draw the automations that have nothing to do with it. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
It used to be a blank box. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
An automation with nothing new to do still sets off what waits for it, which read as if nothing happened. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
It blamed the automation's parameters, which hold no sensor references: its flex config can. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Waiting for an automation gave 403 for another organisation's and 422 for an unknown ID, which told which IDs exist. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Flix6x
left a comment
There was a problem hiding this comment.
Thanks Mohamed! I went through #2706 on an isolated instance (own Postgres, Redis and worker), at 30ecb5199: the CLI, the Automations page and the new endpoints, with a chain of meter snap → solar forecast → battery schedule ← price check, plus a battery report. Waiting works well across crons and timezones, a killed job is now noticed and fails its run with RQ's reason, and allow-failure does what it says. Some changes I'd like to see before merging, mostly in the scheduling edge cases and the API names, as those are hard to change once released.
Needs a change
-
Reactivating an upstream automation, or changing its cron string or timezone, gets what waits for it skipped until the upstream's next due time.
update_automationresets the upstream's cursor to about now;_state_without_upstream_runthen finds the upstream's latest due time behind its cursor without a run, calls itpassed-over, and the waiting run is skipped. For a daily upstream that is up to a day of skipped runs. A deactivated upstream counts asnot-scheduled(go ahead), so reactivating is worse than leaving it off. A runner outage has the same effect: missed upstream runs are merged into the latest one, so a run waiting for an earlier due time finds it passed over.
Proposal: "passed over" should only mean the upstream really did not run for that due time. A due time from before the upstream's last cursor reset counts as not scheduled (theschedule_revisionbump can tell), and after an outage the waiting run waits for the merged catch-up run instead of being skipped. -
Run with ascendants on an inactive automation leaves its own run waiting for ever. Its ascendants run, but
release_waiting_automation_runreturns early for an inactive automation, and the runner only looks at waiting runs of active ones. The docs say inactive automations can be run on demand. On reactivating, the run went ahead with the time it was due, 9 minutes late. The same holds for a run that is waiting when its automation is deactivated. -
One dependent that the caller may not manage makes Run with dependents fail for the owner. Declaring a dependency needs only read access, but
_trigger_automation_chainchecksmanage-automationson every dependent and answers 403 for the whole request. So another account that can read your automation (e.g. on a public asset) can block your Run with dependents, and you cannot remove their dependency. The run plan meanwhile lists such a dependent as "won't run". Please leave it out of the run, as the ascendants direction does. -
API names, before they are released (renaming later needs a deprecation period, as #2857 now does for
schedule-revision):with-ascendantsis not the opposite ofwith-dependents, and dependencies vs dependents is too easily confused. Please usewith-upstream/with-downstream, as pipeline tools (Airflow, Dagster, dbt) do.run-plan?direction=→run-plan?include=upstream|downstream: runs always flow downstream; the parameter only says which part of the chain is included. The docs can say the included automations run in chain order.leave-out→exclude, matching the next point.- In the run plan, list in
stepsonly what will run (nowill-runflag), and the others in anexcludedlist with their reason ("It is inactive.", "You may not run this automation.", "You left it out."). They still run on their own schedule, just not because of this request. GET /assets/automations?scope=returns at most 500 withtruncated: true, and no way to get the rest.GET /assetsandGET /accountsalready takepage/per-page; please do the same. Also, the cut at 500 comes before dropping unreadable automations, so a caller can get fewer than 500, marked truncated, while more readable ones exist.- Return
waited-for-byon the details endpoint only with?include=waited-for-by: unlikedepends-on, it is derived from other automations, costs a query across them, and can list IDs in accounts the caller cannot read.depends-onstays, as it is what was written.
(depends-onas a list of objects, and the same 422 for unreadable and unknown IDs, look good to me.)
Suggestions
- Run labels in the UI: "Run up to here" and "Run from here", with tooltips saying the automation chain runs up to / from this automation.
- Say why a dependency was passed over.
not-scheduledand "Skipped, as the run(s) of automation(s) 2 (run 4) that this run waits for did not succeed or did not happen." are hard to read. Name the automation and the outcome, e.g. "Skipped: Solar forecast (run 4) failed." or "3: no run due yet (created after 13:00)". Going ahead when the upstream had no run due is the right default, I think. - Help start a new chain: when a dependency is saved on an upstream that has never succeeded, say so and offer to run the chain up to here (don't run it automatically).
- The run plan's "The rest goes by the latest results of those that won't run" is hard to parse, and is said even when there are no results. For each excluded automation, show when it last succeeded, and when never: "Check day-ahead prices has never run successfully, so Battery schedule will likely fail." Warn, don't block.
run-automationsprints "No automations due" in a tick where it failed a killed run and skipped the run waiting for it. An operator watching the cron log sees nothing. Print one line per reconciled or released run, with the run record's text; "No automations due" only when nothing happened.- "Automations on this page" vs "Automations on this asset only": with include sub-assets on, "this page" means this asset and the ones below it; off, the two are the same. Name them by what they contain ("This asset and the assets below it" / "This asset only").
- Allow-failure in the edit form: one box is fine, but its mixed state looks unticked. A line under it would help, e.g. "Differs per automation it waits for (Solar forecast: yes; Check day-ahead prices: no). Ticking or unticking sets all of them."
flexmeasures show automations --id: "Depends on" (automations) sits next to "Reads from" (sensors) and they read alike; show each dependency's cron string, and maybe head the sensor lists "Sensors it reads" / "Sensors it writes". It also lists no "Waited for by", which the API has.- Minor: run plan steps are not in dependency order for diamonds (A waits for B and C, C waits for B: C is listed before B), while the docs say "in the order they run"; execution is correct.
The chain drawing could use some work (an arrow running behind a box, both arrowheads on one spot, cut-off names), but that can be a follow-up issue.
Questions (from reading the code, not reproduced)
- On-demand runs left
claimed(the API process dies between committing the runs and dispatching them) seem never to be retried, as only scheduled runs are, so their chain would wait for ever. Is that covered somewhere? - Downstream runs are released and dispatched inside the upstream's job (
record_outcome_on_automation_run), so that work counts against the upstream job's timeout. Intended?
Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
with-ascendants and with-dependents become with-upstream and with-downstream, leave-out becomes exclude, and run-plan takes include=upstream|downstream instead of direction, in the API, the CLI and the Automations page. The run plan lists only what runs under steps, and the rest under excluded with the reason, which now includes "You left it out.". Both directions share one chain function, so exclude works downstream too, a trigger lists every run under runs, and the steps come in chain order, also for a diamond. The page asks the server for the plan again when a branch is ticked, rather than working it out itself. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…using the chain Waiting for an automation only takes reading it, so another organisation could keep its owner from running Run with downstream, with a 403, and the owner could not remove that dependency. Such an automation is now excluded, along with what can only be reached through it, as upstream already did. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…at 500 GET /assets/automations takes page and per-page, like GET /assets, and reports num-records with a page; without page, it lists them all. What the caller may not read is dropped before paging, so a page holds as many automations as asked for while there are more. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Unlike depends-on, it is derived from other automations, costs a query across them, and can name IDs in organisations the caller cannot read. The Automations page asks for it before deleting an automation, which is what it is for. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ught up with Reactivating an upstream automation, or giving it another cron string or timezone, reset its cursor to about now, so a run waiting for one of its earlier due times found it passed over and was skipped, for up to a day with a daily upstream. An automation now records when its current revision started, and a due time before it counts as not scheduled, so what waits for it goes ahead. After a runner outage, the missed upstream runs are coalesced into the latest one, which a run waiting for an earlier one now waits for, rather than being skipped. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
… runs on deactivation Running an inactive automation with what is upstream of it ran its upstream automations, but left its own run waiting for ever, as neither the release nor the runner looked at waiting runs of inactive automations. A run on demand is now released whether its automation is active or not. Deactivating an automation cancels its scheduled runs that are still waiting, which would otherwise go ahead late on reactivation, and what waits for them goes ahead without them, as for an inactive automation. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Each upstream run a run waits for gets a summary, such as "Solar forecast (automation 3, run 4) failed." or "... was not due yet at 2026-08-05 13:00 UTC, as it was created after that.", in the times of the upstream automation's own clock. The skip reason and the waiting message are made of these, rather than bare IDs and states, and the skip reason names only the runs it was skipped for. The Automations page shows the summaries of the latest run. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Their tooltips say the automation chain runs up to, or from, this automation, as do the titles of the plans they show first. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ver did The run plan's excluded automations come with the steps that wait for them, and with when they last succeeded, as those steps go by the results they produced then. The plan shown before a chain runs says so per excluded automation, rather than "The rest goes by the latest results of those that won't run", and warns where one never succeeded, as what waits for it will likely fail, without keeping the chain from running. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ns due" A tick that failed a killed job's run, and skipped or set off the runs waiting for it, printed "No automations due", so an operator watching the cron log saw nothing. The runner's tick now reports each run it ended for a job that ended without saying so, and each run that released as a result, in that order, with the run's own message, and "No automations due" only when nothing happened. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
"Automations on this page" and "Automations on this asset only" become "This asset and the assets below it" and "This asset only", and without the automations of the assets below it listed, which makes the two the same, only "This asset only" is offered. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Where the automations it waits for differ, the box's mixed state looked unticked, so a line under it names each of them with yes or no, and says that ticking or unticking sets all of them. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…its sensors flexmeasures show automations --id said "Depends on" for automations right next to "Reads from" for sensors, which read alike. It now says "Waits for", with each one's recurrence and timezone, as those decide which of their runs are waited for, adds "Waited for by", and heads the sensor lists "Sensors it reads" and "Sensors it writes". Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
A run on demand left claimed, such as by an API process that went down between recording a chain of runs and queueing the first, was never retried, as only scheduled runs are, and never counted as given up, so its chain waited for ever. The runner now fails such a run once the claim on it expires, which skips what waits for it, and reports it. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ng what waits for its run A job releases the runs waiting for its run once it recorded its outcome, which counts against its own timeout. Running out of time there had the queue's exception handler fail the job after all, and with it the run, so that what still waited for it was skipped. A recorded success now stands, and a runner releases what the job did not get to on its next tick. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
|
@Flix6x 1–3 fixed in bdf5188, e75a9af and 9262fb7; 4 in adbe42f, 1803aca and bcccc3b, with your names (with-upstream/with-downstream, include, exclude, steps/excluded, paging, ?include=waited-for-by). |
Description
An automation can now wait for the automations whose results it uses, instead of relying on staggered cron strings.
For example, a data pull feeds a report, which feeds a forecast.
allow-failure).--with-dependents,?with-dependents=true) re-runs a whole chain, for example after a backfill.--with-ascendants,?with-ascendants=true) brings one automation up to date: what it waits for runs first, it runs last, and what waits for it does not run.run-planendpoint. Branches above an automation can be left out there (leave-out), and what will not run says why, such as an automation on a public asset.GET /assets/automations?scope=public|all, which lists what you may read, 500 at most.waited-for-by.depends-on, CLI--depends-on/--allow-failure-of/--no-dependencies, and a UI Waits for list of tick boxesname,assetandasset-nameof each upstream automation the caller may read. One they may not read stays a bare ID.automation_dependencytable,waiting/skippedrun states, and a runtrigger(scheduleoron-demand)documentation/changelog.rstdocumentation/cli/change_log.rstdocumentation/api/change_log.rstUnlike the approach on the issue, runs wait in the database rather than through RQ's
depends_on, for two reasons:depends_onpassed through.A waiting run is released as soon as its upstream run ends.
A job reports its own outcome, which it cannot do when it is killed, so the runner asks Redis about the jobs that are still unfinished.
RQ fails a job whose work-horse died, and one whose worker disappeared once its heartbeat lapsed.
A job that left Redis without reporting counts as failed, and a job that is still queued or running is left alone, however long it takes.
A chain run on demand records a run per automation, so its runs can wait for each other.
They wait only for the runs started with them, and never for scheduled ones.
Running with ascendants can have several starting points, one per branch, and each starts right away.
An automation that does not run takes along what is above it, as nothing would use those results, and what waits for it goes by the results it already has.
No permission changes here: running needs permission to manage automations, so a normal user sees a public automation as one that won't run.
The unique index on runs now covers scheduled runs only, so an on-demand run can share a minute with a scheduled one.
The runner does not retry on-demand runs.
Look & Feel
How to test
pytest flexmeasures/data/tests/test_automation_dependencies_fresh_db.py flexmeasures/api/v3_0/tests/test_automation_dependencies_api_fresh_db.py flexmeasures/ui/tests/js/test_automation_chain.py, plus the dependency tests incli/tests/test_automations.pyandtest_asset_copy_automations_fresh_db.py.kill -9the work-horse of a running job, and runflexmeasures jobs run-automations: its run fails with the reason, and what waits for it is released.Further Improvements
skippedruns.Related Items
Closes #2586. Part of #2557.
Sign-off